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A  standard  strategy  in  simulation,  for  cooq>aring  two  stochastic  systems, 
is  to  use  a  common  sequence  of  random  numbers  to  drive  both  systems.  Certain 
theoretical  and  methodological  results  require  that  the  coupled  system  be 
regenerative.  It  is  shown  that  if  the  stochastic  systems  are  Markov  chains 
with  countable  state  space,  then  the  coupled  system  is  necessarily 
regenerative.  An  example  is  given  which  shows  that  the  regenerative  property 
can  fail  to  hold  in  general  state  space,  even  if  the  individual  systems  are 
regenerative. 
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SIGNIFICANCE  AND  EXPLANATION 


Suppose  that  a  simulator  wishes  to  compare  the  steady-state  performance 
of  two  stochastic  systems*  A  natural  way  to  do  this  is  to  subject  each  of  the 
systems  to  the  same  sequence  of  random  disturbances.  In  other  words,  one 
drives  each  system  with  the  same  random  inputs.  This  is  known,  in  the 
simulation  literature,  as  the  method  of  common  random  numbers.  One  of  the 
most  powerful  tools  available  for  analyzing  steady-state  simulation  problems 
requires  that  the  process  being  simulated  be  regenerative.  Roughly  speaking, 
this  means  that  the  process,  when  viewed  on  an  appropriate  random  time  scale, 
behaves  like  a  sequence  of  independent  and  identically  distributed  random 
variables.  In  this  paper,  we  relate  the  above  concepts  by  obtaining 
conditions  under  which  a  process  simulated  via  common  random  numbers  is 
regenerative. 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  report. 


REGENERATIVE  STRUCTURE  OP  NARXOV  CHAIMS  SIMULATED  VIA 
COMMON  RANDOM  NUMBERS 

Peter  N.  Glynn 

1.  introduction 

For  i  “  1,...,a,  lot  ■  (X^n)  t  n  »  0}  bo  a  Markov  choin  on  stato  apace  Ei. 
Assuming  that  the  X^'a  corroapond  to  ■  altarnativo  deaigna  for  a  now  ayatem,  one  would 
like  to  compare  the  a  deaigna-  To  be  precise,  one  would  like  to  deteraine  the  chain  Xr 
which  minimizea  r^  »  Ef^(X^)  over  1  <  i  <  a,  where  the  f ^tx^ ) *s  are  real-valued  coat 
functionala.  Simulator a,  when  faced  with  thie  problem,  frequently  uae  the  method  of  common 
randoa  number a.  In  other  words,  rather  than  aiaulating  the  a  chaina  independently,  one 
drivea  all  a  chaina  uaing  the  aaae  aequence  of  randoa  inputa.  Since  each  chain  ia 
subjected  to  the  aaae  randoa  diaturbancea,  one  hopea  that  the  atatiatical  efficiency  of  the 
comparison  procedure  will  be  improved,  through  a  variance  reduction  of  aoae  aort. 

HEIDELBERG ER  and  IGLEHART  (1979)  atudied  thia  problea,  in  the  caae  that  the  f1(X1)'a 
take  the  fora 

1  n_1 

fi<xl>  -  “  l  W*))  * 

n*"  k-0 

for  aaae  function  i  Ij  ■*  It.  They  ahowed  that  when  a  -  2,  efficiency  of  the 
aiaulation  experiaent  ia  indeed  improved  when  the  X^ ' a  and  g^'s  are  auitably  aonotone. 
Their  argument  required  that  the  paired  proceaa  (Xj,X2)  be  a  poaitive  recurrent  regenera¬ 
tive  aequencei  no  verifiable  conditione  guaranteeing  thia  regenerative  property  were  given, 
however.  In  thia  note,  we  ahull  ahow  that  if  Ej,...,Ea  are  countable  and  if  the  X^'a 
are  poaitive  recurrent  chaina,  then  the  joint  proceaa  (X^,...,Xm)  is  regenerative.  He 
conclude  with  an  example  which  ahowa  that  the  result  can  fail  if  the  E^*s  are 


uncountable 


2.  The  Main  jwglt 


Our  main  result  will  require  that  the  E^'s  be  countable.  Thus,  without  loss  of 
generality,  we  aay  assuae  that  E^  _c  ■  {0,1,...}  for  1  <  i  <  a.  Let 
pi  ”  (Pi<3,k)  »  J»k  e  E^)  be  the  transition  matrix  for  the  Markov  chain  X1#  and  let 

“  (it.(j)  s  j  e  B^)  be  the  associated  initial  distribution,  k  Markovian  coupling  of  the 

processes  X,,...,]^  is  a  Markov  chain  X  ■  Cx(n)  t  n  >  0}  satisfying! 

(2.1)  i.)  for  n  >  0,  X(n)  i  (X( 1,n) , . . . ,X(a,n) )  C  B  2  E1  x...x  EB, 

ii.)  for  1  <  1  <  m,  P{X(i ,0)  -  j}  -  u^j),  j  €  B1# 

iii.)  for  1  <  1  <  a,  n  >  0, 

p{X(i,n+1)  -  k  |  X(i,n)  -  j}  -  Pid.k),  j,k  e  tL. 

The  method  of  common  random  numbers,  as  practiced  in  simulation,  leads  naturally  to 
Markovian  couplings. 

(2.2)  Bxass»le.  In  general,  Harkov  chains  are  simulated  through  recursions  of  the  fora 

XA(k+1)  -  hi(X1(k),  ei(k-H)) 

where  -  {{^(k)  i  k  >  1}  is  a  sequence  of  independent  and  identically  distributed 
(i.i.d. )  random  variables  (r.v.'s).  For  exasple,  if  the  process  is  generated  through 
inversion,  then  the  ^(kl's  are  uniform  on  [0,1]  i  see  p.  807  of  [4]  for  details.  In 
any  case,  if  the  1  4  1  <  m,  have  a  common  distribution,  as  occurs  through  inversion, 

one  can  use  the  stream  of  random  numbers  t  1  to  gsnerate  all  a  chains  via  the  recursions 
X(i,k+1)  *  hi(X(i,k),  C1(k+1))i  this  Markovian  coupling  is  the  msthod  of  common  random 
numbers. 

(2.3)  Example .  Suppose  that  Pj  -  P2  is  an  irreducible  positive  recurrent  transition 

matrix  with  Invariant  probability  v.  If  Wj(j)~*(j)  and  ie  arbitrary,  one  is 

interested  in  studying,  for  a  given  Markovian  coupling,  various  properties  of  the  random 
time  T(D)  •  inf{n  >  0  i  x(n)  e  D},  where  D  is  the  ’diagonal"  set  {(x,y)  e  t,  «  Bj  i 
x  «  y}.  The  "coupling  time"  T(D)  provides  information  on  the  rate  at  which  X2(n) 


O  * .*  *.*  v‘*>  *  V  .  •  «i,‘ 

aNa'a  .  *  j>  V  -S  aN 


s' 
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converge*  to  the  invariant  probability  * ;  aee  PITMAN  (1974),  for  example .  The  eoat 
frequently  used  coupling,  in  this  area  of  atudy,  is  the  independent  coupling,  in  which 
p{x(n)  -  (k,i)  |  X(n-1)  -  (i,j»)  -  p1(i,k)p2(J,i) 

p{X(0)  -  (k,t )}  -  v(k)|iU) 

for  i,  j,  k,  l  e  I1 . 

Let  P  “  (p(x,y)  :  x,  y  e  ■)  be  the  transition  matrix  of  X  and  v  the  initial 
distribution  of  X.  The  state  space  K  may  be  partitioned  into  subsets  T,  Cf,  C2, — 
where  T  consists  of  transient  states  and  the  Cj/e  are  irreducible  closed  seta  of 
recurrent  statesi  see  $XNLAR  (1975),  p.  125  -  131  for  details, 

(2.4)  Theorem.  Suppose  are  irreducible  positive  recurrent  transition 

matrices,  with  invariant  probability  distributions  v^,...,*^.  Then, 

m 

i.)  C  -  u  Ci  ^  0  consists  of  positive  recurrent  states, 
i-1 

ii. )  ?{T(C)  <  •}  »  1,  where  T(C)  -  inf{n  >  0  t  X(n)  e  c), 

iii.)  for  j  >  1,  the  invariant  probability  distribution  »( j»*  >  concentrated  on 
Cj  satisfies 

iKji*,  x...x  x  (k>  x  *i+1  x...x  ^)  -  »t(k),  k  e  It,  1  <  i  <  m. 

This  theorem  shows  that  for  any  Markovian  coupling  of  countable  state  Markov  chains, 
the  joint  process  X  is  eventually  absorbed  into  some  closed  irreducible  subset  Cj  upon 
which  the  chain  is  positive  recurrent.  Furthermore,  regardless  of  the  class  Cj  into 
which  the  chain  X  is  absorbed,  the  marginal  distributions  of  the  invariant  probability 
associated  with  Cj  will  be  precisely  thoee  of  the  original  chains  X^,...,XB. 

Proof  of  the  theorem.  Let  X  have  an  arbitrary  initial  distribution  w,  and  consider  the 
probability  measures  defined  by 


t^CJ  -  ^  l  P{X(k)  e  •} 
k-0 


i  n“ 1 

QJi.*)  -  ±  I  *{X<i.k>  e  *)  • 
n  n 


Sine*  has  invariant  probability  by  assuaption,  standard  Markov  chain  theory,  plus 

(2.1)  iii.),  shows  that  Qn(i,«)  — >  »A(«)  in  the  discrete  topology  on  Ei  (see 
p.  16  of  BILLINGSLEY  (1968)),  where  “->  denotes  weak  convergence.  Thus,  (Q  ( i , *  )  i 
n  >  1)  is  tight  (see  Theorea  6.2  of  [2])  i.e.,  for  any  e  >  0,  there  exists  a  cowpact  set 

-  R1(e)  (consisting  of  finitely  many  points)  such  that  Qn(i,Ki(e ) )  >  1  -  e/a.  Set 

a 

K  -  K1  x. . .x  tg  and  observe  that  K  «  n  *...»  *i-i  x  *i  x  *...*  E^.  Thus,  for 

any  n  >  1, 


Qn(K) 


^(RC) 


X...X 


“i-1XKlX*i+1 


X  »  e  #X 


V 


>  i  -  l  v,  *'*'x  “i-i^i^i+i  *•••*  V 


1  -  l  Q  (i,R?)  >  1  -  e  . 
i-1  "  1 


Since  K  has  finitely  many  points,  and  is  therefore  cospact  in  the  discrete  topology,  it 
follows  that  {Q  (•)  s  n  >  1}  is  tight.  Thus,  by  Theorea  6.1  of  [2],  we  are  guaranteed 


the  existence  of  a  subsequence  and  a  probability  »( • )  such  that 


V1 


0  (•)  -  l  P{X(j)  e  •}  — >  »(•) 

k  k  J-O 


as  k  ♦  -.  Select  x  e  K  so  that  «({x})  >  0.  Since  Q^liy})  ♦  0  for  transient  or  null 


recurrent  states  y,  it  aust  be  that  x  is  positive  recurrent,  so  that  C  ?  Further¬ 
more  ,  by  concentrating  |i  on  Cj,  it  follows  that  Cj  contains  a  positive  recurrent 
state,  so  that  evidently  Cj  aust  have  all  states  positive  recurrent  (see  Theorea  3.16  of 


[31  )• 


For  (2.4)  ii.),  note  that 


f. 


_ .  K 

p{t_  <  •}  >  lim  —  l  p{*(j)  e  c) 

c  j-0 


l  »({x}>  -  1  , 

xec 


since  our  discussion  above  shows  that  *  aust  be  concentrated  on  positive  recurrent 
states.  For  the  last  assertion,  let  u  be  the  Invariant  probability  «(js*)  con¬ 
centrated  on  Cj  end  observe  that 


Kjl*,  *...*  Ei_1  *  Ck}  *  Ei  x...*  En) 

-  (^(E,  *...x  E^,  x  {k}  x  ka  . In) 

“  Qjjd.tk) )  ♦  s^k)  > 


the  first  equality  is  by  invariance  of  i(}r),  and  the  convergence  follows  frost  (2.1) 
iii. ) . 

(2.3)  Example  (continued).  Suppose  that  Pj  »  P2  is  an  aperiodic  irreducible  positive 
transition  matrix.  Then,  under  the  independent  coupling,  the  state  space  E  of  X  is 
easily  seen  to  be  irreducible.  By  Theorem  2.4,  it  follows  that  X  has  one  positive 
recurrent  irreducible  class  of  states  C. 

h  natural  question  to  ask  at  this  point  is  whether  the  set  of  transient  states  T  can 
be  non-empty.  The  following  example  shows  that  this  is  in  fact  possible. 

(2.S)  Example.  Let  m  *  2  in  Example  2.2,  and  suppose  that  h^  :  X*  *  [0,1]  ♦  E* 

satisfies: 

i.)  p{h.|(i,U)  -  1)  -  0  ,  i  ¥  0 

ii.)  p{h2(i,U)  -  1}  -  0  ,  i  2 

iii.)  hf(0,x)  *  1  if  and  only  if  x  >  1/2 

iv.)  h2(2,x)  -1  if  and  only  if  x  <  1/2, 

where  U  is  a  uniform  r.v.  and  the  h^'s  are  mappings  under  which  the  X^'s  are 
irreducible  and  positive  recurrent.  For  i  "  1,2,  set  X(i,0)  -  1  and  put  X(i,kt1)  - 


V  ■-  •-  V  ^  ■  •  •  •  ^  ...  -  .  • 


Ha  claim 


h1(X1(k),U)(4.1)>  where  {u^  :  k  >  1}  ia  a  sequence  of  i.i.d.  uniform  r.v.'a. 
that  for  n  >  1, 

p{x(n)  -  (1.1)  |  X(0)  -  (1,1)>  -  0  , 

which  clearly  implies  that  (1.1)  is  transient  for  X.  Note  that  by  (2.4)  i.)  -  ii.). 
X(n)  -  (1,1)  only  if  X(n-1)  -  (0,2).  But 

p{x(n)  -  (1,1)  |  X(n-1 )  -  (0,2)} 

-  P{h, (0,U  )  -  1  -  h-(2,U  )}  -  0 
in  <  n 

by  (2.4)  iii. )  -  iv.). 

This  has  implications  for  the  regenerative  method  of  simulation  (see  IGLEHAKT  (197B) 
for  details),  as  applied  to  steady  state  analysis  of  countable  state  Markov  chains 
generated  via  common  random  numbers.  By  Example  2.S,  the  simulation  may  start  in  a 
transient  state  if  the  initial  distribution  of  X  is  not  chosen  carefully.  However, 
Theorem  2.4  shows  that  X  ia  eventually  absorbed  into  a  closed  positive  recurrent 
irreducible  class  Cj.  Once  X  is  in  Cj,  the  standard  regenerative  method  may  be 
applied,  using  any  state  in  Cj  as  "regeneration  state."  The  difficulty,  of  course,  is 
determining  precisely  when  X  has  entered  an  absorbing  recurrent  class. 


3.  A  Nonreceneratlve  Markovian  Coupling 


We  will  produce  a  nonregenerative  process  X  “  (X(1,*),  X(2,*)),  in  which  X  is 
obtained  from  regenerative  component  processes  via  the  method  of  common  random  numbers.  To 
be  precise,  we  shall  choose  X(1,*)  and  X(2,*)  to  be  Harkov  chains,  having  identical 
transition  functions,  which  are  individually  positive  recurrent  regenerative  sequences 
(positive  recurrence  shall  mean  here  that  the  expected  time  between  regenerations  is 
finite),  for  any  initial  condition.  However,  we  will  show  that  if  X(1,0)  +  X(2,0),  the 
joint  chain  X  is  not  a  positive  recurrent  regenerative  sequence. 

Let  e  ■  R  and  put  h(x,y)  “  (V x)  +  y.  Assume  that  P(5(k+1)  e  dx)  «  f(x)dx, 
i  ”  1,2,  where: 

i . )  f  is  continuous  and  positive  on  R 
ii.)  a  -  e|C (k+1) |  <  •  . 

The  concept  of  a  Markovian  coupling  generalises,  in  the  obvious  way,  to  Harkov  chains 
taking  values  in  R. 

(3.1)  Proposition.  The  Harkov  chain  Z  defined  by  Z(k+1)  -  h(Z(k),  5 (k+ 1 ) )  is  a 
positive  recurrent  regenerative  sequence. 

Proof.  We  will  prove  that  there  exists  a  set  A,  a  positive  number  X,  and  a  probability 
measure  9  on  E  such  that: 

a. )  PlT(A)  <  «  |  2(0)  -  z}  -  1  for  ell 

z  c  E,  where  T( A)  “  inftn  >  1  :  Z(n)  ex}. 

b. )  P{z(1)  £  •  |  2(0)  -  *}  >  X*C) 

for  all  z  e  A. 

Conditions  a. )  and  b. )  allow  one  to  use  a  "splitting  technique"  due  to  ATHREYA  and  MEY 
(1978)  and  NUHHELIN  (1978)  to  construct  regeneration  tines  for  Z,  under  any  initial 
condition. 

Let  A  •  {z  :  jz|  <  2(a+1)}i  we  use  a  "test  function"  criterion  due  to  TWEEDIE  (1976) 
to  verify  a.).  Note  that  for  z  f.  A,  and  k(«)  -  |*|, 

E{k(Z(  1 ) )  |  Z(0)  -  z)  -  E|(V2  )z  +  5(1) | 

<  (V2  )|z|  +  a  <  2a  +  1  <  k(z)  -  1 


-7- 


from  which  it  follow*  (see  Theorem  6.1  of  (9]}  that  e{t(A)  |  Z(0)  ■  z)  <  k(z)  <  m  for 
i  /  A,  proving  a.). 

For  b. ),  lot  X  -  2  min{f(z)  :|z|  <  a+2),  sty)  ■  1/2  on  (“1,1]  and  zero  elsewhere, 
and  observe  that 

p{z<1)  e  b  |  z(0)  -  z}  -  P{(V2  )z  +  5(1)  e  b> 

“  J 3  “  <V*2  >  *  /6  s(y)dy  2  X$(B) 

for  z  e  A.  For  the  positive  recurrence,  note  that  our  above  bound  on  the  expected  hitting 
time  of  A  shows  that 

sup  e{t(A)  |  Z(0)  -  z)  <  sup  E{k(Z(1) )  |  Z<0)  -  z)  +  1 
ZCA  ZCA 

<  sup  |c|  +  a  +  1  <  m  • 

ZCA 

Since  A  is  compact  with  positive  ^-measure,  and  the  transition  function  of  Z  is  weakly 
continuous,  it  follows  that  A  is  a  status  set  for  Z;  the  above  bound  then  proves  that 
Z  has  an  invariant  probability  measure.  (See  Proposition  5.4  and  Theorem  9.1  of  [9]  for 
results  and  definitions.)  Theorem  6.1  of  (1]  yields  the  positive  recurrence  as  a 
consequence. 

Define  X(i,«)  via  the  recursion  X(i,k+1)  “  (V2  )X(i,k)  ♦  5<k+1).  Then, 

k  i  k 

X(i,k)  -  l  23  5( j)  *  2_kX(i,0>  . 

j-1 

Clearly,  X(n)  converges  weakly  to  Y  -  (Y1,Y2)  where  Y1  -  Y2  a.s.  Suppose  that  X  is 
a  positive  recurrent  regenerative  sequence  with  regeneration  times  Tj,  T2,...  .  Then, 

.  n-1  V1 

-  I  p{x(k)  C  B>  ♦  E(  l  I(X(k)  C  B)}  =  W(B) 
k-0  k-T1 

E{T2  -  T,} 

by  SMITH  (1954),  p.  27.  It  is  easily  verified  that  the  set  function  w ( • )  must  be  a 
probability  measure,  and  thus  *(•)  -  p{y  e  •}.  Hence,  setting  B  *■  { (x^ , ) :  x1  -  Xj} 


in  (3.2)  yields 


1  n"1 

0  -  -J-  l  ?{X( 1 ,k)  -  X(2,k)}  /  PiY,  -  Yj)  -  1  , 

k“0 

if  X( 1 ,0 )  f  X( 2 ,0 ) .  This  contradiction  shows  that  X  is  not  a  positive  recurrent 
regenerative  sequence. 
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